perf(gfql): engage #1658 index for seeded gfql()/Cypher chains on ALL 4 engines (incl. typed edges)#1734
Merged
Merged
Conversation
lmeyerov
force-pushed
the
feat/gfql-typed-edge-index-coverage
branch
from
July 19, 2026 17:25
5f18f4c to
31b07ba
Compare
Contributor
Author
|
Stacked: rebased onto #1735 ( |
This was referenced Jul 19, 2026
lmeyerov
force-pushed
the
feat/gfql-typed-edge-index-coverage
branch
from
July 19, 2026 20:29
31b07ba to
13d2442
Compare
lmeyerov
marked this pull request as ready for review
July 19, 2026 20:32
lmeyerov
commented
Jul 19, 2026
lmeyerov
force-pushed
the
feat/gfql-typed-edge-index-coverage
branch
from
July 19, 2026 20:45
13d2442 to
8df8249
Compare
…das/cuDF (incl. typed edges) A seeded hop expressed as a gfql()/Cypher chain (not a direct g.hop) always scanned on pandas/cuDF even with a resident index: _chain_impl attaches a synthetic per-edge id column (copy(deep=False)+assign) -> a fresh edge-frame object -> the index's `source_ref is df` identity guard missed. - registry.rebind_edges: re-point edge adjacency indexes at the augmented frame (safe: shallow copy preserves src/dst by value; fingerprint unchanged by an added column). Called in chain.py right after the edge-index augmentation. - Typed edges: relax _hop_is_index_coverable for simple scalar-equality edge_match on the wavefront path; index_seeded_hop applies the edge predicate to CSR-matched rows each hop, parity-exact with the scan's filter_edges_by_dict. - Predicate/membership edge_match, edge_query, and the direct-hop path stay on scan. Measured (dgx, 500k/4M/4-types, warm median, index==scan): pandas typed 2.1x / untyped 2.7x; cuDF typed 1.9x / untyped 2.0x. polars/polars-gpu untyped already engage via the native lazy executor; typed polars is a tracked follow-up. 27 new 4-engine differential + engagement tests; existing 109-test index suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…edges The native polars lazy chain executor (lazy/engine/polars/chain.py) attaches its synthetic edge id via with_row_index -> a fresh edge frame -> the #1658 identity guard missed, so typed-edge chains scanned on polars/polars-gpu (untyped already engaged). Apply the same rebind_edges fix at the polars augmentation site. Now ALL FOUR engines engage the index for typed AND untyped seeded chain/Cypher hops. Measured (dgx, 500k/4M/4-types, index==scan): polars typed 1.3x / untyped 5.4x; polars-gpu typed engaged / untyped 12.4x. Broadened the engagement test to all engines; 138 index tests pass (109 existing + 29 new), 4-engine. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…y check; cast ArrayLike operands for mask & Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
…nd_edges (review) Review feedback on #1734: - types.py gains ScalarMatchValue / EdgeMatchValue / EdgeMatch / SimpleEqualityEdgeMatch mirroring exactly the runtime shapes filter_by_dict accepts; is_simple_equality_edge_match becomes a TypeGuard so the index path gets real narrowing instead of Optional[dict], and _build_edge_keep_mask / index_seeded_hop take EdgeMatch. - rebind_edges no longer trusts its docstring: the strong-ref swap is gated on the index's O(1) structural fingerprint (row count + cols + engine) and indexed-column presence; any mismatch DROPS the edge index (safe miss -> scan) instead of re-pointing it at a frame it wasn't built over. Value-level preservation stays the caller's promise (checking it would be the O(E) scan this path avoids). + enforcement test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
lmeyerov
force-pushed
the
feat/gfql-typed-edge-index-coverage
branch
from
July 19, 2026 21:08
8df8249 to
11d0bab
Compare
… scan path (review blockers) Review of #1734 found the guard's 'simple scalar equality' narrower than filter_by_dict's runtime behavior — three defects in one seam, all fixed: - F1 (BLOCKER): frozenset (+ pd.Index/pd.Series) leaked past the (list, tuple, set, dict) check but are membership (isin) on the scan path; a bare == is silently all-False -> wrong answer on public g.hop(). The guard now delegates to filter_by_dict._is_membership_filter_value (single source of truth); EdgeMatchValue widened to match. - F2 (BLOCKER): null-carrying match columns (pandas nullable dtypes, polars nulls — common after NaN->null coercion) produced NA cells whose to_numpy() yields an object-dtype array, exploding later at rows[edge_keep[rows]] (IndexError). Masks now fill_null(False)/fillna(False) before materializing — parity-exact (null == val drops on the scan path too). - F3 (IMPORTANT): numeric-col-vs-str / string-col-vs-numeric edge_match returned a silent empty subgraph where the scan raises (GFQLSchemaError E302 on pandas/cuDF; polars its own ComputeError). The mask builder mirrors filter_by_dict's exact two checks and declines -> falls back to the scan -> users get the SAME error as their engine's scan. + 3 engine-parametrized regression tests (frozenset parity, null-column no-crash+parity, dtype-mismatch error parity). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A seeded hop expressed as a
gfql([...])/Cypher chain (rather than a directg.hop(...)) previously always fell back to the O(E) scan, even with a resident #1658 adjacency index. Root cause: both chain executors attach a synthetic per-edge id column to the edge frame (pandas/cuDF eager_chain_impl:copy(deep=False)+assign; polars native lazy executor:with_row_index) → a fresh edge-frame object → the index'ssource_ref is dfidentity guard missed →index_seeded_hopreturnedNone→ scan. This is the material Search lever for real Cypher/LDBC workloads (seeded traversals expressed as chains with typed edges).Changes
registry.rebind_edges(new_edges)— re-points the edge adjacency indexes' identity guard at the chain's augmented frame. Safe: the shallow copy /with_row_indexpreserves the indexedsrc/dstcolumns by value, and the structural fingerprint (rows + bound columns + engine) is unchanged by an added column. NODE_ID left untouched. Applied at both augmentation sites:compute/chain.py(pandas/cuDF eager) andlazy/engine/polars/chain.py(polars/polars-gpu native lazy)._hop_is_index_coverablenow accepts a simple scalar-equalityedge_match(-[:KNOWS]->/e_forward({"type": X})) on the wavefront path;index_seeded_hopapplies the edge predicate to the CSR-matched rows each hop, parity-exact with the scan'sfilter_edges_by_dict.edge_match,edge_query, and the direct-hop (non-wavefront) path.Validation (dgx-spark, 500k nodes / 4M edges / 4 edge-types, warm median-of-7; every cell verified index result == scan result)
All shapes PARITY=True (exact). All four engines now engage the index for typed AND untyped seeded chains. Small synthetic-graph typed speedups understate the real win (the reverse-pass still scans and synthetic degree is low); the index-vs-scan gap is far larger on real LDBC/Pokec shapes. Regression:
test_index.py138 passed (109 existing + 29 new differential/engagement/rebind tests), 4-engine.🤖 Generated with Claude Code